Projecting an image of a real object

ABSTRACT

In some examples, a projection capture system comprises a controller, a camera operatively connected to the controller for capturing images of an object on a work surface of a workspace, a projector operatively connected to the controller, and a mirror above the projector to reflect light from the projector onto the work surface. The camera is located higher than the projector, and the controller is to control the camera to capture an image of a real object on the work surface, and control the projector to project the image of the real object onto the work surface.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. application Ser. No. 14/130,943, filedJan. 6, 2014, which is a national stage application under 35 U.S.C. §371of of PCT/US2011/058896, filed 2 Nov. 2011, which claims priority fromPCT/US2011/053947, filed 29 Sep. 2011; PCT/US2011/046253, filed 2 Aug.2011; and PCT/US2011/045983, filed 29 Jul. 2011, which are all herebyincorporated by reference in their entirety.

BACKGROUND

Various types of mixed reality systems have been developed to producenew environments where real and virtual objects co-exist and interact inreal time. Virtual whiteboard and other types of remote collaborationsystems have also been developed to enable remote users to share andmanipulate information simultaneously at multiple locations.

DRAWINGS

FIGS. 1A and 1B are perspective, exterior views illustrating one exampleof a new projection capture system. In FIG. 1A, the image of a twodimensional object (a hardcopy photograph) has been captured anddisplayed. In FIG. 1B, the image of a three dimensional object (a cube)has been captured and displayed.

FIG. 2 is a perspective, interior view illustrating one example a newprojection capture system.

FIG. 3 is a block diagram of the projection capture system shown in FIG.2.

FIG. 4 is block diagram illustrating one example of a user input devicein the system shown in FIGS. 2 and 3.

FIGS. 5 and 6 are side and front elevation views, respectively,illustrating the positioning of the camera and the projector in theprojection capture system shown in FIGS. 2 and 3.

FIGS. 7-11 are a progression of side elevation views showing variouspositions for the projector and the camera in a projection capturesystem, illustrating some of the problems associated with moving theglare spot out of the camera capture area.

FIGS. 12 and 13 illustrate one example of the camera in the projectioncapture system shown in FIGS. 2 and 3.

FIG. 14 illustrates one example of the projector in the projectioncapture system shown in FIGS. 2 and 3.

FIGS. 15 and 16 illustrate examples of the user input device in theprojection capture system shown in FIGS. 2 and 3.

FIGS. 17-19 are perspective views illustrating one example of a newportable projection capture device.

FIGS. 20-22 illustrate three example scenarios for using a projectioncapture system with other devices.

FIG. 23 is a block diagram illustrating one example of a new projectioncapture device that includes object recognition and audio/videoteleconferencing capabilities.

FIG. 24 is a block diagram illustrating one example architecture forimplementing a projection capture device such as the one shown in FIG.23 in a collaborative environment.

FIG. 25 is a block diagram illustrating one example of a controller forimplementing a layering technique in which real and virtual objects aretreated as visually interchangeable logical layers.

FIG. 26 shows a projection capture system in which real checkers arepositioned on a virtual checkerboard projected on to the work surface.

FIG. 27 illustrates one example of Z axis layering for the real andvirtual objects in the system of FIG. 26.

FIG. 28 is a flow chart illustrating one example method for implementinga layering technique in which real and virtual objects are treated asvisually interchangeable logical layers.

The same part numbers designate the same or similar parts throughout thefigures.

DESCRIPTION

The examples shown in the figures and described below illustrate but donot limit the invention, which is defined in the Claims following thisDescription.

A new projection capture system has been developed to improve theinteractive user experience working with real objects and projectedobjects on a physical work surface and to improve virtual collaborationamong multiple remote users. The new system may be implemented, forexample, in one or more stand-along portable devices deployed on anordinary work surface. A digital camera, projector and controlprogramming are housed together in a desktop unit that enables aprojection augmented virtual reality in which real and projected/virtualobjects can be manipulated and shared simultaneously among multipleremote users. Such portable devices can be deployed almost anywhere atany time for interactive collaboration across a comparativelyinexpensive platform suitable not only for larger, enterprise businessenvironments but also for small businesses and even personal consumers.

As used in this document, a “real” object means an object that is notdisplayed, projected or otherwise rendered as an image; and a “virtual”object means an object that is displayed, projected or otherwiserendered as an image.

Examples of a new projection capture system and portable projectioncapture devices will be described first with reference to FIGS. 1-19.Examples of the implementation of the new projection capture system anddevices in a collaborative environment will then be described withreference to FIGS. 20-28.

Projection Capture System and Devices

FIGS. 1A and 1B are perspective, exterior views illustrating one exampleof a new projection capture system 10 and an interactive workspace 12associated with system 10. FIG. 2 is a perspective view illustrating oneexample of a projection capture system 10 with exterior housing 13removed. FIG. 3 is a block diagram of system 10 shown in FIG. 2.Referring to FIGS. 1A, 1B, 2, and 3, projection capture system 10includes a digital camera 14, a projector 16, and a controller 18.Camera 14 and projector 16 are operatively connected to controller 18for camera 14 capturing an image of an object 20 in workspace 12 and forprojector 16 projecting the object image 22 into workspace 12 and, insome examples, for camera 14 capturing an image of the projected objectimage 22. The lower part of housing 13 includes a transparent window 21over projector 16 (and infrared camera 30).

In the example shown in FIG. 1A, a two dimensional object 20 (a hardcopyphotograph) placed onto a work surface 24 in workspace 12 has beenphotographed by camera 14 (FIG. 2), object 20 removed to the side ofworkspace 12, and object image 22 projected onto a work surface 24 whereit can be photographed by camera 14 (FIG. 2) and/or otherwisemanipulated by a user and re-projected into workspace 12. In the exampleshown in FIG. 1B, a three dimensional object 20 (a cube) placed ontowork surface 24 has been photographed by camera 14 (FIG. 2), object 20removed to the side of workspace 12, and object image 22 projected intoworkspace 12 where it can be photographed by camera 12 and/or otherwisemanipulated by a user and re-projected into workspace 12.

In one example implementation for system 10, controller 18 is programmedand projector 16 is configured to project object image 22 into the sameposition in workspace 24 as the position of object 20 when its image wascaptured by camera 14. Thus, a one-to-one scale digital duplicate 22 ofan object 20 can be projected over the original allowing a digitalduplicate in its place to be manipulated, moved, and otherwise alteredas desired by a local user or by multiple remote users collaborating inthe same projected workspace 12. The projected image can also be shiftedaway from the original, allowing a user to work with the original andthe duplicate together in the same workspace 12.

System 10 also includes a user input device 26 that allows the user tointeract with system 10. A user may interact with object 20 and/orobject image 22 in workspace 12 through input device 26, object image 22transmitted to other workspaces 12 on remote systems 10 (not shown) forcollaborative user interaction, and, if desired, object image 22 maybephotographed by camera 14 and re-projected into local and/or remoteworkspaces 12 for further user interaction. In FIG. 1A, work surface 24is part of the desktop or other underlying support structure 23. In FIG.1B, work surface 24 is on a portable mat 25 that may include touchsensitive areas. In FIG. 1A, for example, a user control panel 27 isprojected on to work surface 24 while in FIG. 1B control panel 27 may beembedded in a touch sensitive area of mat 25. Similarly, an A4, letteror other standard size document placement area 29 may be projected ontowork surface 24 in FIG. 1A or printed on a mat 25 in FIG. 1B. Of course,other configurations for work surface 24 are possible. For example, itmay be desirable in some applications for system 10 to use an otherwiseblank mat 25 to control the color, texture, or other characteristics ofwork surface 24, and thus control panel 27 and document placement area29 may be projected on to the blank mat 25 in FIG. 1B just as they areprojected on to the desktop 23 in FIG. 1A.

In the example shown in FIG. 4, user input device 26 includes aninfrared digital stylus 28 and an infrared camera 30 for detectingstylus 28 in workspace 12. Although any suitable user input device maybe used, a digital stylus has the advantage of allowing input in threedimensions, including along work surface 24, without a sensing pad orother special surface. Thus, system 10 can be used on a greater varietyof work surfaces 24. Also, the usually horizontal orientation of worksurface 24 makes it useful for many common tasks. The ability to usetraditional writing instruments on work surface 24 is advantageous oververtical or mobile computing interfaces. Projecting an interactivedisplay on to a working desktop mixes computing tasks with the standardobjects that may exist on a real desktop, thus physical objects cancoexist with projected objects. As such, the comfort of using realwriting instruments as well as their digital counterparts (like stylus28) is an effective use model. A three-dimensional pad-free digitalstylus enables annotation on top of or next to physical objects withouthaving a sensing pad get in the way of using traditional instruments onwork surface 24.

In one example implementation for system 10, projector 16 serves as thelight source for camera 14. Camera capture area 32 (FIG. 12) andprojector display area 34 (FIG. 14) overlap on work surface 24. Thus, asubstantial operating efficiency can be gained using projector 16 bothfor projecting images and for camera lighting. The light path fromprojector 16 through workspace 12 to work surface 24 should bepositioned with respect to camera 14 to enable user display interactionwith minimal shadow occlusion while avoiding specular glare off worksurface 24 and objects in workspace 12 that would otherwise blind camera14. The system configuration described below avoids the glare inducedartifacts that would result from a conventional camera lighting geometrywhile still maintaining a sufficiently steep incident angle for theprojector light path desired for proper illumination and projection oftwo and three dimensional objects in workspace 12.

Ideally, projector 16 would be mounted directly over workspace 12 at aninfinite height above work surface 24 to insure parallel light rays.This configuration, of course, is not realistic. Even if projector 16was moved down to a realistic height above work surface 24 (but stillpointing straight down), the projector's light would be reflected offglossy and semi-glossy surfaces and objects straight back into camera14, creating a blinding specular glare. Thus, the glare spot must bemoved out of camera capture area 32. (Specular glare refers to glarefrom specular reflection in which the angle of incidence of the incidentlight ray and the angle of reflection of the reflected light ray areequal and the incident, reflected, and normal directions are coplanar.)

To achieve a commercially reasonable solution to this problem ofspecular glare, camera 14 and projector 16 are shifted away from thecenter of capture and display areas 32, 34 and projector 16 ispositioned low, near base 36, as shown in FIGS. 5 and 6, and a foldmirror 38 is introduced into the projector's light path to simulate aprojector position high above work surface 24. The simulated position ofprojector 16 and the corresponding light path above mirror 38 are shownin phantom lines in FIGS. 5 and 6. However, before describing theconfiguration shown in FIGS. 5 and 6 in more detail, it is helpful toconsider the problems associated with other possible configurations formoving the glare spot out of camera capture area 32.

In FIG. 7, camera 14 is positioned at the center of capture area 32 withan overhead projector 16 slightly off center so that camera 14 does notblock the projector light path. In the configuration of FIG. 7, thespecular glare spot 39 (at the intersection of incident light ray 41 andreflected light ray 43) falls within capture area 32 and, thus, willblind camera 14 to some objects and images in capture area 32. Inaddition, for the configuration shown in FIG. 7, where camera 14 andprojector 16 are both positioned high above the base, system 10 would betop heavy and, thus, not desirable for a commercial productimplementation. If projector 16 is positioned to the side the distanceneeded to move glare spot 39 out of camera capture area 32, as shown inFIG. 8, the corresponding projector lens offset required would not befeasible. Also, any product implementation for the configuration ofsystem 10 shown in FIG. 8 would be undesirably broad and top heavy.

Moving camera 14 off center over capture area 32 brings projector 16 into make the system less broad, as shown in FIG. 9, but the projectorlens offset is still too great and the product still top heavy. In theconfiguration shown in FIG. 10, projector 16 is raised to a height sothat it may be brought in close enough for an acceptable lens offsetbut, of course, the product is now too tall and top heavy. The mostdesirable solution is a “folded” light path for projector 16, shown inFIGS. 5 and 11, in which the “high and tight” configuration of FIG. 10is simulated using fold mirror 38. In FIGS. 5 and 11, projector 16 andthe upper light path are folded over the reflecting surface of mirror 38to project the same light path on to work surface 24 as in theconfiguration of FIG. 10. This folding effect is best seen in FIG. 5where fold angles Θ1=θ2 and φ1=φ2.

As shown in FIGS. 5 and 6, camera 14 is placed in front of the mirror 38over workspace 12 so that it does not block the projector's light path.Camera 14 is positioned off center in the Y direction (FIG. 5) as partof the overall geometry to keep glare spot 39 out of capture area 32with an acceptable offset for both camera 14 and projector 16. Projector16 is focused on mirror 38 so that light from projector 16 is reflectedoff mirror 38 into workspace 12. By moving projector 16 down low andintroducing a fold mirror 38 into the projector light path, glare spot39 is kept out of capture area 32 with an acceptable projector offsetand system 10 is sufficiently narrow, short and stable (not top heavy)to support a commercially attractive product implementation.

Thus, and referring again to FIGS. 1A, 1B, and 2, the components ofsystem 10 may be housed together as a single device 40. Referring alsoto FIG. 3, to help implement system 10 as an integrated standalonedevice 40, controller 18 may include a processor 42, a memory 44, and aninput/output 46 housed together in device 40. Input/out 46 allows device40 to receive information from and send information to an externaldevice, as described below with reference to FIGS. 20-22. Whileinput/output 46 is shown in FIG. 3 as being part of controller 18, someor all of input/output 46 could be separate from controller 18.

For the configuration of controller 18 shown in FIG. 3, the systemprogramming to control and coordinate the functions of camera 14 andprojector 16 may reside substantially on controller memory 44 forexecution by processor 42, thus enabling a standalone device 40 andreducing the need for special programming of camera 14 and projector 16.Programming for controller 18 may be implemented in any suitable form ofprocessor executable medium including one or more software modules,hardware modules, special-purpose hardware (e.g., application specifichardware, application specific integrated circuits (ASICs), embeddedcontrollers, hardwired circuitry, etc.), or some combination of these.Also, while other configurations are possible, for example wherecontroller 18 is formed in whole or in part using a computer or serverremote from camera 14 and projector 16, a compact standalone appliancesuch as device 40 shown in FIGS. 1A, 1B and 2 offers the user fullfunctionality in an integrated, compact mobile device 40.

Referring now to FIG. 12, camera 14 is positioned in front of mirror 38above workspace 12 at a location offset from the center of capture area32. As noted above, this offset position for camera 14 helps avoidspecular glare when photographing objects in workspace 12 withoutblocking the light path of projector 16. While camera 14 representsgenerally any suitable digital camera for selectively capturing stilland video images in workspace 12, it is expected that a high resolutiondigital camera will be used in most applications for system 10. A “highresolution” digital camera as used in this document means a camerahaving a sensor array of at least 12 megapixels. Lower resolutioncameras may be acceptable for some basic scan and copy functions, butresolutions below 12 megapixels currently are not adequate to generate adigital image sufficiently detailed for a full range of manipulative andcollaborative functions. Small size, high quality digital cameras withhigh resolution sensors are now quite common and commercially availablefrom a variety of camera makers. A high resolution sensor paired withthe high performance digital signal processing (DSP) chips available inmany digital cameras affords sufficiently fast image processing times,for example a click-to-preview time of less than a second, to deliveracceptable performance for most system 10 applications.

Referring now also to FIG. 13, in the example shown, camera sensor 50 isoriented in a plane parallel to the plane of work surface 24 and lightis focused on sensor 50 through a shift lens 52. This configuration forsensor 50 and lens 52 may be used to correct keystone distortionoptically, without digital keystone correction in the object image. Thefield of view of camera 14 defines a three dimensional capture space 51in work space 12 within which camera 14 can effectively capture images.Capture space 51 is bounded in the X and Y dimensions by camera capturearea 32 on work surface 24. Lens 52 may be optimized for a fixeddistance, fixed focus, and fixed zoom corresponding to capture space 51.

Referring to FIG. 14, projector 16 is positioned near base 36 outsideprojector display area 34 and focused on mirror 38 so that light fromprojector 16 is reflected off mirror 38 into workspace 12. Projector 16and mirror 38 define a three dimensional display space 53 in workspace12 within which projector 16 can effectively display images. Projectordisplay space 53 overlaps camera capture space 51 (FIG. 12) and isbounded in the X and Y dimensions by display area 34 on work surface 24.While projector 16 represents generally any suitable light projector,the compact size and power efficiency of an LED or laser based DLP(digital light processing) projector will be desirable for mostapplications of system 10. Projector 16 may also employ a shift lens toallow for complete optical keystone correction in the projected image.As noted above, the use of mirror 38 increases the length of theprojector's effective light path, mimicking an overhead placement ofprojector 16, while still allowing a commercially reasonable height foran integrated, standalone device 40.

One example of suitable characteristics for system 10 as a standalonedevice 40 are set out in Table 1. (Dimension references in Table 1 areto FIGS. 5 and 6.)

TABLE 1 CAMERA PROJECTOR Sensor Mpixel 12 Mp Sensor aspect 1.333 ratioX/Y Pixel size .00175 mm CX Object full size X 427 mm PX IllumFull-field X 310 mm CY Object full size Y 320 mm PY Illum Full-field Y310 mm CH Camera height 450 mm PH Projector height 670 mm CS Camerashift in Y 150 mm PS Projector shift in Y 330 mm Magnification⁻¹ 66Sensor pixels X 4016 Lens offset 216% Sensor pixels Y 3016 Lens shift108% Sensor size X 7.028 mm Max Y-fan angle 35.76 deg Sensor size Y5.278 mm Min Y-fan angle 14.84 deg Image size X 6.470 mm Half-field X203.5 mm Image size Y 4.848 mm Half-field Y 482.5 mm Half-field X 213.5mm Throw ratio 1.65 Half-field Y 280 mm Max throw angle 38.01 degFull-field angle 76.08 deg CC Camera clearance 51.6 mm distance Sampling220 ppi GC Glare spot clearance 44.4 mm resolution distance Capturelength X 464.85 mm Capture length Y 348.35 mm

Since projector 16 acts as the light source for camera 12 for still andvideo capture, the projector light must be bright enough to swamp outany ambient light that might cause defects from specular glare. It hasbeen determined that a projector light 200 lumens or greater will besufficiently bright to swamp out ambient light for the typical desktopapplication for system 10 and device 40. For video capture and real-timevideo collaboration, projector 16 shines white light into workspace 12to illuminate object(s) 20. For an LED projector 16, the time sequencingof the red, green, and blue LED's that make up the white light aresynchronized with the video frame rate of camera 14. The refresh rate ofprojector 16 and each LED sub-frame refresh period should be an integralnumber of the camera's exposure time for each captured frame to avoid“rainbow banding” and other unwanted effects in the video image. Also,the camera's video frame rate should be synchronized with the frequencyof any ambient fluorescent lighting that typically flickers at twice theAC line frequency (e.g., 120 Hz for a 60 Hz AC power line). An ambientlight sensor can be used to sense the ambient light frequency and adjustthe video frame rate for camera 14 accordingly. For still image capture,the projector's red, green, and blue LED's can be turned onsimultaneously for the camera flash to increase light brightness inworkspace 12, helping swamp out ambient light and allowing fastershutter speeds and/or smaller apertures to reduce noise in the image.

The example configuration for system 10 integrated into a standalonedevice 40 shown in the figures and described above achieves a desirablebalance among product size, performance, usability, and cost. The foldedlight path for projector 16 reduces the height of device 40 whilemaintaining an effective placement of the projector high above workspace12 to prevent specular glare in the capture area of camera 12. Theprojector's light path shines on a horizontal work surface 24 at a steepangle enabling 3D object image capture. This combination of a longerlight path and steep angle minimizes the light fall off across thecapture area to maximize the light uniformity for camera flash. Inaddition, the folded light path enables the placement of projector 16near base 36 for product stability.

Suitable input devices and techniques for use in system 10 include, forexample, finger touch, touch gestures, stylus, in-air gestures, voicerecognition, head tracking and eye tracking. A touch pad can be used toenable a multi-touch interface for navigating a graphical user interfaceor performing intuitive gesture actions like push, flick, swipe, scroll,pinch-to-zoom, and two-finger-rotate. Depth cameras using structuredlight, time-of-flight, disturbed light pattern, or stereoscopic visionmight also be used to enable in-air gesturing or limited touch and touchgesture detection without a touch pad. A touch-free digital stylus isparticularly well suited as a user input 26 for system 10. Thus, in theexample shown in the figures, user input 26 includes an infrared digitalstylus 28 and an infrared camera 30 for detecting stylus 28 in workspace12. As noted above, a touch-free digital stylus has the advantage ofallowing input in three dimensions, including along work surface 24,without a sensing pad or other special surface.

Referring now to FIGS. 4 and 15, input device 26 includes infraredstylus 28, infrared camera 30 and a stylus charging dock 54. Stylus 28includes an infrared light 56, a touch sensitive nib switch 58 to turnon and off light 56 automatically based on touch, and a manual on/offswitch 60 to manually turn on and off light 56. (Nib switch 58 andmanual switch 60 are shown in the block diagram of FIG. 4.) Light 56 maybe positioned, for example, in the tip of stylus 28 as shown in FIG. 15to help maintain a clear line-of-sight between camera 30 and light 56.Light 56 may also emit visible light to help the user determine if thelight is on or off.

Nib switch 58 may be touch sensitive to about 2 gr of force, forexample, to simulate a traditional writing instrument. When the stylus'snib touches work surface 24 or another object, nib switch 58 detects thecontact and turns on light 56. Light 56 turning on is detected by camera30 which signals a touch contact event (similar to a mouse button clickor a finger touch on a touch pad). Camera 30 continues to signalcontact, tracking any movement of stylus 28, as long as light 56 stayson. The user can slide stylus 28 around on any surface like a pen totrace the surface or to activate control functions. When the stylus nibis no longer in contact with an object, light 56 is switched off andcamera 30 signals no contact. Manual light switch 60 may be used tosignal a non-touching event. For example, when working in a threedimensional workspace 12 the user may wish to modify, alter, orotherwise manipulate a projected image above work surface 24 by manuallysignaling a “virtual” contact event.

Infrared camera 30 and mirror 38 define a three dimensional infraredcapture space 61 in workspace 12 within which infrared camera 30 caneffectively detect light from stylus 28. Capture space 61 is bounded inthe X and Y dimensions by an infrared camera capture area 62 on worksurface 24. In the example shown, as best seen by comparing FIGS. 14 and15, infrared camera capture space 61 is coextensive with projectordisplay space 53. Thus, infrared camera 30 may capture stylus activationanywhere in display space 53.

In one example implementation shown in FIG. 16, camera 30 is integratedinto the projection light path such that the projector field-of-view andthe infrared camera field-of-view are coincident to help make surestylus 28 and thus the tracking signal from infrared camera 30 isproperly aligned with the projector display anywhere in workspace 12.Referring to FIG. 16, visible light 64 generated by red, green and blueLEDs 66, 68, and 70 in projector 16 passes through various optics 72(including a shift lens 74) out to mirror 38 (FIG. 14). Infrared light75 from stylus 28 in workspace 12 reflected off mirror 38 towardprojector 16 is directed to infrared camera sensor 76 by an infraredbeam splitter 78 through a shift lens 80. (Similar to the exampleconfiguration for camera 14 described above, infrared light sensor 76for camera 30 may be oriented in a plane parallel to the plane of worksurface 24 and light focused on sensor 76 through shift lens 80 for fulloptical keystone correction.)

It may be desirable for some commercial implementations to houseprojector 16 and infrared camera 30 together in a single housing 82 asshown in FIG. 16. The geometrical configuration for infrared camera 30shown in FIG. 16 helps insure that the stylus tracking signal is alignedwith the display no matter what height stylus 28 is above work surface24. If the projector field-of-view and the infrared camera field-of-vieware not coincident, it may be difficult to calibrate the stylus trackingat more than one height above work surface 24, creating the risk of aparallax shift between the desired stylus input position and theresultant displayed position.

Although it is expected that workspace 12 usually will include aphysical work surface 24 for supporting an object 20, work space 12could also be implemented as a wholly projected work space without aphysical work surface. In addition, workspace 12 may be implemented as athree dimensional workspace for working with two and three dimensionalobjects or as a two dimensional workspace for working with only twodimensional objects. While the configuration of workspace 12 usuallywill be determined largely by the hardware and programming elements ofsystem 10, the configuration of workspace 12 can also be affected by thecharacteristics of a physical work surface 24. Thus, in some examplesfor system 10 and device 40 it may be appropriate to consider thatworkspace 12 is part of system 10 in the sense that the virtualworkspace accompanies system 10 to be manifested in a physical workspacewhen device 36 is operational, and in other examples it may beappropriate to consider that workspace 12 is not part of system 10.

FIGS. 17-19 are perspective views illustrating another example of aportable projection capture device 40 and an interactive workspace 12associated with device 40. Referring to FIGS. 17-19, portable device 40includes a digital camera 14 for capturing still and video images of anobject 20 in capture area 32 (and in capture space 51) and a projector16 for illuminating an object in capture area 32 (and capture space 51)and for projecting images onto display area 34 (and into a display space53). A two dimensional object 20 (a hardcopy photograph) placed incapture area 32 has been photographed by camera 14 (FIGS. 17 and 18),object 20 removed from capture area 32, and an object image 22 projectedonto display area 34 (FIG. 19) where it can be photographed by camera 14and/or otherwise manipulated by a user.

In this example, device 40 also includes an electronic display 84 forselectively displaying a live feed from camera 14, an image previouslycaptured by camera 14, or the representation of an image as it ismanipulated by the user through a graphical user interface (GUI) 86projected into display space 53. (GUI 86 is projected onto display area32 in the example shown in FIGS. 17-19.) Camera 14, projector 16, anddisplay 84 are operatively connected together through a controller 18and housed together in housing 13 as a single portable device 40.Projector 16 is positioned below camera 14 high in housing 13 to projectlight directly into display space 53 and on to display area 34.Projector display space 53 and display area 34 overlap camera capturespace 51 and capture area 32 so that projector 16 can serve as the lightsource for camera 14 capturing images of real objects 20 in space 51 andon area 32 and so that camera 14 can capture images of images 20projected into space 51 and on area 32.

Controller 18 is programmed to generate and projector 16 projects a GUI86 that includes, for example, device control “buttons” such as Capturebutton 88 in FIGS. 17 and 18 and Undo, Fix, and OK buttons 90, 92, and94, respectively, in FIG. 19. Although device 40 in FIGS. 17-19 mightalso include a more complex GUI and corresponding control programming incontroller 18, as well as other user input device(s), the deviceconfiguration of FIGS. 17-19 illustrates basic digital copying and imagemanipulation functions more suitable for a less expensive consumerdesktop product market.

The examples of system 10 and device 40 shown in the figures, with onecamera 14 and one projector 16, do not preclude the use of two or morecameras 14 and/or two or more projectors 16. Indeed, it may be desirablein some applications for a system 10 and device 40 to include more thanone camera, more than one projector or more than one of other systemcomponents.

Projection Capture in a Collaborative Environment

FIGS. 20-22 illustrate three example scenarios for using a projectioncapture device 40 with other devices. In the use scenario of FIG. 20,projection capture device 40 is linked to a computer workstation 88 anda mobile device 90. In the use scenario of FIG. 21, multiple projectioncapture devices 40 are linked together through a server 92. In the usescenario of FIG. 22, projection capture devices 40, computer workstation88, and mobile device 90 are linked together through a server 92. Eachlink 94 in FIGS. 20-22, represents generally one or more of a cable,wireless, fiber optic, or remote connection via a telecommunicationlink, an infrared link, a radio frequency link, or any other connectoror system that enables electronic communication between the linkeddevices. While individual links 94 are shown, multiple devices mightutilize the same link. Also, other use scenarios are possible. Forexample, multiple projection capture devices 40 could be linked togetherdirectly, without a server 92.

In each of the scenarios illustrated in FIGS. 20-22, individual userscan create, manipulate, transfer, and store virtual objects with devices40, 88, 90 and multiple users can collaborate among devices 40, 88, 90with a mixture of real and virtual objects. Virtual objects include, forexample, digital content rendered as projected images at projectioncapture devices 40 as well as digital content rendered on a display asslides, documents, digital photos and the like at other devices 88 and90. As noted above with reference to FIGS. 1A, 1B, 2, and 3, eachprojection capture device 40 may be configured to project an objectimage 22 into the same position in workspace 24 as the position ofobject 20 when its image was captured by camera 14. Thus, a one-to-onescale digital duplicate 22 of an object 20 can be projected over theoriginal allowing a digital duplicate in its place to be manipulated,moved, and otherwise altered as desired by remote users collaborating inworkspaces 12 projected at devices 40 or displayed at devices 88, 90.Any suitable alteration technique may be used including, for example,touch and gesture recognitions such as “pinch to zoom” or input from IRstylus 28 for a projection device 40 and/or altering a digital file fora computing device 88, 90. The projected image can also be shifted awayfrom the original, allowing a user to work with the original and theduplicate together in the same workspace 12.

FIG. 23 is a block diagram illustrating one example of a projectioncapture device 40 that includes audio and a video teleconferencingfeature 96. Referring to FIG. 23, conferencing feature 96 includes afront facing camera 98, a microphone 100, and a speaker 102. Theaddition of conferencing feature 96 allows device 40 to function as afull featured collaboration tool. Projection capture device 40 in FIG.23 also includes an object recognition device 104 for distinguishingbetween real and virtual objects in the workspace. In the example shown,object recognition device 104 includes an infrared camera 106 and aninfrared light 108. Where workspace camera 14 will see real objects andprojected images (virtual objects), infrared camera 106 will see onlythe real objects. Thus, the video stream (or still image data) fromcameras 14 and 106 may be used to distinguish real objects from virtualobjects in the workspace, for example through programming residing oncontroller 18.

An LED, laser or other suitable infrared light 108 may be used withcamera 106 to illuminate the workspace to improve object recognition.Also, while it may be possible to use the same infrared camera for bothobject recognition (camera 106) and for sensing an IR stylus (camera 30in FIGS. 4, 15, and 16), it is expected that the camera frame rate forobject recognition usually will not need to be as high as the frame ratefor sensing stylus position but may require higher resolution.Consequently, it may be desirable for some implementations to useseparate infrared cameras for object recognition and stylus sensing. IRcamera 106 is just one example of a suitable objection recognitiondevice 104. Other implementations are possible. A depth camera, forexample, could be used in device 40 instead of an IR camera todistinguish between real and virtual objects in the workspace.

FIG. 24 illustrates one example architecture for the programming andsignal processing to implement a projection capture device 40 from FIG.23 in a collaborative environment. In this example, real and virtualobjects are managed separately. Real objects captured with a workspacecamera 14 are managed through motion video while virtual objects aremanaged through still image graphics elements such as bitmap images,vector graphics objects, and text. For a stand-alone projection capturedevice 40, the blocks shown in FIG. 24 are implemented in controller 18.Referring to FIG. 24, the video streams from workspace camera 14 andfront facing camera 98 go to an outgoing video manager 110 where theyare output to linked devices at block 112. Still images from workspacecamera are routed to an object manager 114. Object manager 114 is thesystem component that stores and manipulates digital content, includingcompositing image objects for projector display at block 116 andcompositing image objects for output to linked devices at block 118. Theaudio signals from conferencing microphone 100 go to an outgoing audiomanager 120 where they are output to linked devices at block 112. Inputfrom linked devices at block 122 is routed to the appropriatemanager—still images are routed to object manager 114, video is routedto an incoming video manager 124 for output to object compositing 116and projector 16, and audio is routed to an incoming audio manager 126for output to speaker 102.

One example for managing the interaction between real and virtualobjects will now be described with reference to FIGS. 25-28. In thisexample, real and virtual objects are treated as visuallyinterchangeable logical layers that allow a projection capture device tointerpret and control its workspace and, in a collaborative environment,to help each user interact effectively with local and remote objects.FIG. 25 is a block diagram illustrating one example for implementingthis layering technique through a programming module 128 residing onprojection capture device controller 18. Layering module 128 associatesa real object with one visual layer (or set of layers) and associates avirtual object with another visual layer (or set of layers). As notedabove with reference to FIG. 24, real objects captured with a workspacecamera 14 are managed through digital motion video while virtual objectsare managed through digital still image graphic elements. Object manager114 stores and manipulates digital content, including visual layeringimplemented in layering module 128. In one example, each video and stillimage digital element is associated with a position in an XYZ coordinatesystem. Layering module 128 uses the XYZ position information tocharacterize the relative position of each element with layers in eachplane of the coordinate system—the XY, XZ, and YZ planes. The visualposition of each element may then be altered by manipulated thecorresponding layer(s). For example, where the XY layer for one object(or element in an object) may initially appear to be above the XY layerfor another object, controller 18 may alter this visual positioning bymoving one or both layers in the Z direction.

Reference will now be made to FIGS. 26 and 27 to help illustrate thelayering technique. FIG. 26 shows a projection capture device 40 andworkspace 12 in which real checkers 130 are positioned on a virtualcheckerboard 132 projected on to work surface 24. Workspace 12 alsoincludes virtual checkers 134 projected on to checkerboard 132, forexample using digital content from one or more linked devices 40, 88, 90in FIG. 22. Solid lines indicate real objects in FIG. 26 and dashedlines indicate virtual objects. FIG. 27 illustrates Z axis layering forthe objects in FIG. 26. The Z axis position of double (kinged) checkersis represented by logical layer 136 on top of layer 138. Layer 138represents the position of single checkers on top of layer 140. Layer140 represents the position of virtual checkerboard 132 on top of layer142 which represents the position of work surface 24. Once the positionof an object or an element in an object is associated with a logicallayer, layering module 128 (FIG. 25) maintains state information aboutthe visual order of the layers. As the visual relationships are changed,for example when a checker in FIG. 26 is moved, the layers are reorderedaccording to new position data associated with one or more of thedigital elements.

In general, and referring to the flow chart of FIG. 28 along with theblock diagrams of FIGS. 23 and 25, controller 18 identifies the presenceand location of a real object in workspace 12 at block 202, for exampleusing workspace camera 14 and object recognition device 104. At block204, layering module 128 associates the real object with a first logicallayer (or a first set of logical layers for three dimensional spatialpositioning). At block 206, a virtual object (an object image) isprojected into workspace 12 at a location corresponding to a secondlogical layer. Then, in response to a change in the position of one ofthe objects or in response to some other user input, layering module 128alters the visual adjacency of the logical layers to reflect thechange/input at block 208. For example, the first layer might initiallybe visually adjacent to work surface 24 and the second layer visuallyadjacent to the first layer. In response to a change/input, the order ofthe layers is reversed so that the second layer becomes visuallyadjacent to work surface 24 and the first layer becomes visuallyadjacent the second layer.

The use of a projection capture system 10 (through a portable device 40for example) in a collaborative environment such as that shown in FIGS.20-22, enables each collaborator to see and interact with both local andremote objects. A live video feed among linked devices shares user inputin real time through a variety of different input devices andtechniques. While collaborator interaction may be more comprehensivewhen every participant is using a projection capture device 40 (orsystem 10), an effective collaborative environment is still availableusing different types of devices. Interesting cooperative work ispossible where one participant uses a projection capture system 10 andthe other participants use devices running system 10 client programming.In such “mixed” environments, instead of interacting with a system 10workspace, a participant would use a mouse, keyboard or touchpad tointeract through the client programming. Instead of a projectiondisplay, a client programming on a collaborative device 88 or 90 may usea window display in the computer workspace. And, instead of an overheadcamera, the client application could use screen shots to capture objectimages.

As noted at the beginning of this Description, the examples shown in thefigures and described above illustrate but do not limit the invention.Other examples, embodiments and implementations are possible. Therefore,the foregoing description should not be construed to limit the scope ofthe invention, which is defined in the following claims.

What is claimed is:
 1. A projection capture system, comprising: acontroller; a camera operatively connected to the controller forcapturing images of an object on a work surface of a workspace; aprojector operatively connected to the controller; and a mirror abovethe projector to reflect light from the projector onto the work surface,wherein the camera is located higher than the projector, the controllerto: control the camera to capture an image of a real object on the worksurface, and control the projector to project the image of the realobject onto the work surface.
 2. The projection capture system of claim1, wherein the controller is to control the projector to project theimage of the real object onto the work surface as a one-to-one scaledigital duplicate of the real object.
 3. The projection capture systemof claim 2, wherein the controller is to control the projector toproject the image of the real object onto the work surface at the sameposition occupied by the real object when the image of the real objectwas captured by the camera.
 4. The projection capture system of claim 1,wherein the controller, the camera, the projector, and the mirror arehoused together as a single portable unit.
 5. The projection capturesystem of claim 1, wherein the projector is a light source for thecamera during capture of the images by the camera.
 6. The projectioncapture system of claim 5, wherein the controller is to control theprojector to produce a time sequencing of different color light that issynchronized with a video frame rate of the camera during capture ofvideo images by the camera.
 7. The projection capture system of claim 1,wherein the mirror is to simulate a projector position above the worksurface.
 8. The projection capture system of claim 7, wherein thesimulating of the projector position causes a folding effect where theprojector is folded over a reflecting surface of the mirror to project asame light path onto work surface as if the projector were providedabove the mirror.
 9. The projection capture system of claim 1, whereinthe controller is to: control the camera and the projector to capture animage of the image of the real object as the image of the real object isbeing altered in the workspace including the work surface, and projectthe altered image of the real object into the workspace.
 10. Anon-transitory processor readable storage medium storing instructionsthat upon execution: cause a camera to capture an image of a real objecton a work surface in a first workspace of a projection capture deviceincluding the camera and a projector; and cause the projector to projectthe image of the real object onto the work surface, wherein the causingof the projecting of the image of the real object onto the work surfacecomprises causing the projector to project the image of the real objectto a mirror located above the projector, the mirror to reflect the imageof the real object into the first workspace, and the camera locatedhigher than the projector.
 11. The non-transitory processor readablemedium of claim 10, wherein the projector is a light source for thecamera during capture of the image of the real object by the camera, andwherein the instructions when executed cause a time sequencing ofdifferent color light produced by the projector to be synchronized witha video frame rate of the camera during the capture of the image of thereal object by the camera.
 12. The non-transitory processor readablemedium of claim 10, the image of the real object is projected onto thework surface at the same position occupied by the real object when theimage of the real object was captured.
 13. The non-transitory processorreadable medium of claim 10, wherein the instructions upon executioncause simultaneous projecting of-the image of the real object into aplurality of workspaces including the first workspace.
 14. Thenon-transitory processor readable medium of claim 10, wherein theinstructions upon execution cause a controller to: receive informationrelating to altering of the image of the real object in the firstworkspace; and cause each of multiple projectors to project the alteredimage of the real object simultaneously into each respective workspaceof a plurality of workspaces.
 15. The non-transitory processor readablemedium of claim 10, wherein the mirror is to simulate a projectorposition above the work surface.
 16. The non-transitory processorreadable medium of claim 15, wherein the simulating of the projectorposition causes a folding effect where the projector is folded over areflecting surface of the mirror to project a same light path onto worksurface as if the projector were provided above the mirror.
 17. A methodcomprising: capturing, by a camera, a digital image of a real object ona work surface in a first workspace; and projecting, by a projector, theimage of the real object to a mirror located above the projector, themirror reflecting the image of the real object onto the work surface,wherein the camera is located higher than the projector.
 18. The methodof claim 17, further comprising: causing, by a controller, simultaneousprojecting of the image of the real object into multiple workspacesincluding the first workspace; receiving, by the controller, an alteredobject image as altered in one of the multiple workspaces; and causing,by the controller, simultaneous projecting of the altered object imageinto the multiple workspaces including the first workspace.
 19. Themethod of claim 18, wherein projecting the image of the real object intothe multiple workspaces comprises projecting the image of the realobject as a one-to-one scale digital duplicate of the real object. 20.The method of claim 17, wherein the mirror is to simulate a projectorposition above the work surface.